Sharp Oracle Inequalities in Low Rank Estimation

The paper deals with the problem of penalized empirical risk minimization over a convex set of linear functionals on the space of Hermitian matrices with convex loss and nuclear norm penalty. Such penalization is often used in low rank matrix recovery in the cases when the target function can be well approximated by a linear functional generated by a Hermitian matrix of relatively small rank (comparing with the size of the matrix). Our goal is to prove sharp low rank oracle inequalities that involve the excess risk (the approximation error) with constant equal to one and the random error term with correct dependence on the rank of the oracle. 1 Main Result Let (X,Y ) be a couple, whereX is a random variable in the space Hm ofm×m Hermitian matrices and Y is a random response variable with values in a Borel subset T ⊂ R. Let P be the distribution of (X,Y ) and let Π denote the marginal distribution of X. The goal is to predict Y based on an observation of X. More precisely, let l : T × R 7→ R+ be a measurable loss function. We will assume in what follows that, for all y ∈ T, l(y; ·) is convex. Given a measurable function f : Hm 7→ R (a “prediction rule”), denote (l • f)(x, y) := l(y; f(x)) and define the risk of f as P (l • f) = El(Y ; f(X)). Partially supported by NSF Grants DMS-1207808, DMS-0906880 and CCF-0808863